Method and system for computer-aided patient stratification based on case difficulty

ABSTRACT

When evaluating patient cases to determine complexity thereof, a computer-aided stratification technique is applied to analyze historical patient case diagnoses and correctness thereof in order to calculate a stratification score ( 20 ) for each of a plurality of abnormality types and/or anatomical locations. When a new patient case is received, the computer-aided stratification technique is applied to evaluate the patient case in view of historical data and assign a stratification score thereto. A ranked list ( 21 ) of current patient cases can be generated according to stratification scores, and physician workload can be adjusted as a function thereof so that workload is balanced across physicians and/or according to physician experience level.

The present innovation finds application in medical diagnosistechnologies, particularly with regard to computer-aided diagnosistherein. However, it will be appreciated that the described techniquesmay also find application in other patient diagnosis systems, otherdiagnosis scenarios, other stratification techniques, and the like.

Accurate diagnosis is important for disease management and therapy of apatient. To arrive at an accurate diagnosis, physicians often spend along time reading, studying, and creating recommendations for“difficult” cases, i.e. unusual or complex cases. On the other hand, foreasier diagnosis cases the physician's diagnosis and recommendations forthe next steps can be generated in a very short time. This is especiallytrue for junior physicians, as if they are presented with a difficultcase they often need to request a second opinion of a more seniorcolleague. That is, there can be a significant difference in the amountof time and effort involved for a physician to assess a difficult caseversus an easy case. An example of this is where a radiologist is askedto assess an image where a lesion is clearly visible and clearlymalignant versus a case where the lesion is difficult to see and has amix of malignant and benign characteristics.

In hospital radiology practice, the radiologist typically works througha daily worklist stored in a radiology information system (RIS) orpicture archiving and communication system (PACS) and consisting ofrecently imaged patients. These systems typically do not consider the“difficulty” of a case, but rather the patients are sorted based ontypes of exams only, without the notion that a particular case may bedifficult to diagnose or not. Conventional systems may have only theintelligence to sort and present the cases by imaging modality andspecialty. For example they can sort the cases based on organs (e.g.breast, liver, etc.) and/or imaging modality (CT, X-ray, ultrasound,DCE-MRI, etc.) only.

The present application provides new and improved systems and methodsthat facilitate stratifying patient cases according to potentialdiagnosis difficulty level, which overcome the above-referenced problemsand others.

In accordance with one aspect, a method of ranking patient casesaccording to difficulty level comprises, for each of a plurality ofpatient cases: retrieving from a database an image study for a patient;identifying an abnormality in a patient image included in the imagestudy; analyzing patient demographic and clinical information; andcalculating a computer-aided stratification score for the patient caseas a function of the identified abnormality and the patient demographicand clinical information; The method further comprises outputting aranked list of the patient cases according to the respectivestratification score assigned to each patient case.

According to another aspect, a system that facilitates ranking patientcases according to difficulty level comprises a computer-aidedstratification module comprising a processor adapted to, for each of aplurality of patient cases: retrieve from a database an image study fora patient; identify an abnormality in a patient image included in theimage study; analyze patient demographic and clinical information; andcalculate a computer-aided stratification score for the patient case asa function of the identified abnormality and the patient demographic andclinical information. The processor is further configured to output(e.g., to a user interface, a printer, or the like) a ranked list of thepatient cases according to the respective stratification score assignedto each patient case.

According to another aspect, a computer-readable medium has storedthereon computer-executable instructions for ranking patient casesaccording to difficulty level, the instructions comprising, for each ofa plurality of patient cases: retrieving from a database an image studyfor a patient; identifying an abnormality in a patient image included inthe image study; analyzing patient demographic and clinical information;and calculating and assigning a computer-aided stratification score forthe patient case as a function of the identity of the abnormality andthe patient demographic and clinical information. The instructionsfurther comprise outputting a ranked list of the patient cases accordingto the respective stratification score assigned to each patient case.

One advantage is that physician workload balance is improved.

Another advantage is that difficult diagnoses can be identified foradditional scrutiny.

Still further advantages of the subject innovation will be appreciatedby those of ordinary skill in the art upon reading and understand thefollowing detailed description.

The drawings are only for purposes of illustrating various aspects andare not to be construed as limiting.

FIG. 1 illustrates a system that facilitates performing computer-aidedpatient stratification to sort patients according to case difficulty, inaccordance with one or more aspects described herein.

FIG. 2 illustrates a method for performing computer-aided stratificationof diagnosis difficulty level for a plurality of patient cases usinghistorical diagnosis accuracy data, in accordance with one or moreaspects described herein.

FIG. 3 illustrates a method for performing computer-aided stratificationof diagnosis difficulty level for a plurality of patient cases as afunction of lesion parameter analysis, in accordance with one or moreaspects described herein

FIG. 4 illustrates a method for performing computer-aided stratificationof diagnosis difficulty level for a plurality of patient cases usingcomputer- aided diagnosis (CADx), in accordance with one or more aspectsdescribed herein.

FIG. 5 illustrates a method for performing computer-aided stratificationof diagnosis difficulty level for a plurality of patient cases, inaccordance with one or more aspects described herein.

The described systems and methods overcome the above-mentioned problemsby stratifying patient cases according to a level of difficultyassociated with the diagnoses of the patients. For instance, to maximizeefficiency and accuracy, the assignment of cases to physician worklistsincludes difficulty as a factor. For example, easier cases can beassigned to junior physicians, while more complex cases are reserved formore senior personnel. In another example, a mix of cases may be equallydistributed across different physicians. The current innovation thusfacilitates assessing the difficulty of a case and employing the resultof the assessment to adjust a clinical workflow. For instance, patientcases are assigned to a particular physician not only based on organtype or imaging modality, but also based on diagnosis difficulty level.In another embodiment, an alert is generated if a case is determined tobe highly complex, such as an alert that recommends a second physician'sreview and/or whether the case would be a useful teaching case.

FIG. 1 illustrates a system 10 that facilitates performingcomputer-aided patient stratification to sort patients according to casedifficulty, in accordance with one or more aspects described herein. Thesystem takes as its input a current patient's clinical case that is tobe evaluated for a specific clinical question. The clinical case caninclude patient information or data comprising patient demographicinformation, clinical information, a current imaging study, etc. Adatabase 12 stores patient information, including but not limited todemographic information, e.g. gender, age, ethnicity, etc. The databasealso stores clinical information for each of a plurality of patients,which may include, e.g. family history, medical history, reason for theimaging study, current condition, symptoms, current treatments, riskfactors, etc. Also stored in the database are acquired imaging studiesfor one or more patients, including, e.g. a CT scan, an MM scan, a PETscan, a SPECT scan, an ultrasound scan, an x-ray or the like.

The clinical question may be broadly described as a screening task(e.g., detection of abnormalities, or the like), a diagnosis task (e.g.,characterization of abnormalities as to their nature and/or malignancy),or an evaluation task (e.g., measurements, assessment of diseaseprogression and/or treatment efficacy). The question may be narrowedfurther by specifying location(s) in the image for evaluation, such asan organ in which abnormalities are being search (e.g. search for breastlesions) or a specific tumor that is being assessed. This informationcan be included in the metadata associated with the patient or imageinformation (such as in a private DICOM field or as acomputer-interpretable segment of a clinical note).

The system further comprises a processor 14 that executes, and a memory16 stores, computer-executable instructions for performing the variousfunctions, methods, techniques, applications, etc., described herein.The memory 16 may be a computer-readable medium on which a controlprogram is stored, such as a disk, hard drive, or the like. Common formsof computer-readable media include, for example, floppy disks, flexibledisks, hard disks, magnetic tape, or any other magnetic storage medium,CD-ROM, DVD, or any other optical medium, RAM, ROM, PROM, EPROM,FLASH-EPROM, variants thereof, other memory chip or cartridge, or anyother tangible medium from which the processor 14 can read and execute.In this context, the system 10 may be implemented on or as one or moregeneral purpose computers, special purpose computer(s), a programmedmicroprocessor or microcontroller and peripheral integrated circuitelements, an ASIC or other integrated circuit, a digital signalprocessor, a hardwired electronic or logic circuit such as a discreteelement circuit, a programmable logic device such as a PLD, PLA, FPGA,Graphics processing unit (GPU), or PAL, or the like.

A computer-aided stratification (CAS) module 18 is executed on theclinical case data 12 to generate a stratification score 20. The scoremay be numerical (such as 0-100) or categorical (such as “easy”,“moderate”, and “difficult”). In one embodiment the CAS module uses theimaging data to generate the stratification score. In other embodimentthe CAS module also uses the demographics and other non-imaginginformation, as is described above. The computer-aided stratificationmodule 18 generates the stratification score, which is used to sort thepatient case based on the predicted difficulty of the case with respectto the clinical question. In another embodiment, the CAS module 18computes a stratification score that assesses the difficulty incharacterizing a given lesion. The CAS module 18 outputs a rankedpatient list 21, which can be ranked according to the stratificationscores (e.g., level of diagnosis difficulty) associated with respectivepatient cases. The patient list can also include, e.g., alertsrecommending a second physician's review (e.g., a second opinion) forspecified patient cases, alerts recommending that a particular case beused as a teaching case or the like, etc. The stratification scores orranking for each patient case can be used, e.g., to ensure thatdifficult cases are assigned to senior physicians, to ensure that anexcessive number of difficult cases is not assigned to any one physician(to balance workload across physicians), etc.

With continued reference to FIG. 1, FIG. 2 illustrates a method forperforming computer-aided stratification of diagnosis difficulty levelfor a plurality of patient cases, in accordance with one or more aspectsdescribed herein. At 50, a database is collected (e.g., a priori)comprising, e.g., patient demographic information, clinical information,imaging study information, etc., and over a large number of cases. Thedatabase also includes the diagnostic assessment of a reading(diagnosing) radiologist, e.g. regarding whether the specified lesion ismalignant or benign. The database further comprises the actual diagnosisas determined by pathology or other adjunct measures, such as stabilityover time (suggestive of benign processes). Thus, the database includesdata indicative of both the radiologist assessment and whether or notthe radiologist's assessment was correct. At 52, using machine learningtechniques, a computer classifier 22 performs a mathematicaltransformation that renders the data in the database into a numericalmeasure of determine likelihood that the physician's diagnosticassessment (e.g., whether a tumor is malignant or benign) will match thetruth (i.e., the actual correct diagnosis as determined by pathology orthe like). That is, CAS module predicts whether a radiologist is likelyto assess the case correctly or incorrectly. In one example, historicaldiagnosis accuracy over a number of patient cases is used to assignstratification scores to current patient cases. For instance, the CASmodule determines whether a particular type of lesion is misdiagnosed ata rate above a predetermined threshold, and assigns that particular typeof lesion a “difficult” rating or score. To further this example, aradiologist may regularly determine that a particular tumor or tumorlocation is malignant and order a biopsy procedure. If the biopsyregularly indicates that the tumor is benign, the particular tumor typeor location (or other tumor metric) can be associated with astratification score of “difficult”. It will be appreciated that theforegoing example is not limited to misdiagnoses as a metric forassigning the stratification score, but rather other metrics may beemployed, such as characteristics of the images or the computed metricsdescribing the level of uncertainty expressed in radiology reports. At54, when applied to a new case for which the specific lesion location(or type or other metric) is pre-specified, as previously described, thescore generated by the classifier becomes the stratification score. Inone embodiment, the CAS algorithm can provide a stratification score foreach individual radiologist based on the diagnostic accuracy of theparticular physician.

With continued reference to FIG. 1, FIG. 3 illustrates a method forperforming computer-aided stratification of diagnosis difficulty levelfor a plurality of patient cases, in accordance with one or more aspectsdescribed herein. For instance, the CAS module 18 computes thestratification score that assesses the difficulty in measuring a givenlesion. In a case where the specific lesion to be measured ispre-specified, the location of the lesion is taken as an inputparameter, at 60. A segmentation algorithm 24 for segmenting a lesion onan image is executed by the processor on the specified lesion location,at 62. This produces a lesion outline or mask 26 which can be used tocompute lesion parameters 28 such as surface area, volume, long-axis,short-axis, or similar common clinical measurements, at 64. A pluralityof alternative lesion outlines (and consequently, alternative masks) canbe computed, at 66, for example as follows. The input location may beperturbed, for example, by randomly sampling from a neighborhood with aset distribution, such as 1 or 2 mm standard deviation. Alternatively, aplurality of different segmentation algorithms may be run with the sameinput, again resulting in a variety of measurements. At 68, thestratification score is derived directly from the variance across thedifferent measurements. The stratification score indicates an estimateof how different radiologists or algorithms would differ in theirassessment of the particular lesion. For example, if a small change toone or more input parameters results in a large change in thesegmentation results, then the case can be assigned a “difficult”stratification score. On the other hand, if large variances in inputvalues result in minimal differences in output values, then the case canbe assigned an “easy” stratification score. In one embodiment, theactual radiologist workflow may be entirely manual. In anotherembodiment, the automated segmentation is employed to estimate theuncertainty and compute the stratification score.

With continued reference to FIG. 1, FIG. 4 illustrates a method forperforming computer-aided stratification of diagnosis difficulty levelfor a plurality of patient cases, in accordance with one or more aspectsdescribed herein. The CAS module 18 computes the stratification scorethat assesses the difficulty in characterizing a given lesion. At 80,the specific lesion to be characterized is pre-specified, and thelocation of the lesion is taken as an input parameter. A computer-aideddiagnosis (CADx) algorithm 30 is executed by the processor to computethe probability of malignancy of the lesion based on image information,at 82. In another embodiment, the CADx algorithm can use imageinformation and patient demographic and clinical information provided at83, when determining the probability of malignancy of the lesion at 82.Techniques for CADx may include, e.g., those that produce scores between0-100, with 100 denoting the highest probability of malignancy. At 84, astratification score can then be derived based directly on the CADxscore. A very high (or very low) CADx score (e.g. <20 or >80) cancorrespond to “easy” cases. “Moderate” and “difficult” cases cansimilarly be mapped.

The CAS module stratification score output is used to affect theworkflow of case reading by clinicians in various manners, at 86. Forinstance, the cases can be further sorted and assigned to the physiciansbased on the physicians' experience. For example, the most difficultcases can be assigned to senior physicians with a certain number ofyears of experience, while the moderately difficult cases are assignedto the less-experienced physicians. In another embodiment, cases can besorted and assigned to physicians in order to reduce the variation indifficulty-level across physicians, i.e. for each physician, defining aworkload metric by computing the sum of the stratification scores overall of the given physician's cases for the day, and then selecting adistribution that minimizes the cross-physician variance in thisworkload metric.

In another embodiment, cases can be sorted within a single physician'sworklist in order to distribute difficult cases evenly across the day,for example, by again defining a workload metric and then selecting adistribution that minimizes the hour-to-hour (or other time scale)variance in this workload metric for a single physician. In anotherembodiment, an indicator can be placed next to patients within aworklist (e.g. on a RIS), indicating the complexity of each patient'scase. The indicator may be a flag based on a threshold, i.e. for casesabove a certain level of complexity, or a numerical value, or a visualindicator such as a color flag, graphical line, or the like indicativeof that value.

According to another example, exceptionally difficult cases can beflagged for automatic double reading, i.e. reading by a secondradiologist. This can be implemented by setting a threshold that exceedsa threshold above which this event is triggered. In another embodiment,difficult cases can be flagged for possible inclusion in a teaching fileor as a case study.

FIG. 5 illustrates a method for performing computer-aided stratificationof diagnosis difficulty level for a plurality of patient cases, inaccordance with one or more aspects described herein. At 100, an imagingstudy is generated or retrieved from a database of a particular patient.The imaging study may be of any suitable imaging modality, such as MM,CT, ultrasound, x-ray, nuclear, etc. At 102, an abnormality (e.g., alesion) is identified. In one embodiment, the abnormality is identifiedusing computer-aided detection. In another embodiment, the abnormalityis manually annotated or identified. At 104, patient demographicinformation and/or clinical information are retrieved. At 106, acomputer-aided stratification score is calculated as a function of thepatient demographic information, the clinical information and the lesionidentity, or any combination of the foregoing information. That is, thescore can be calculated using CADx, based on image segmentation data, orthe like. Acts 100, 102, 104, and 106 are iteratively performed for eachof a plurality of patient cases. At 108, patient cases are sortedaccording to their stratification scores.

According to an example, a breast dynamic contrast enhanced(DCE)-magnetic resonance imaging (MRI) screening study is performed on a43-year old woman. The patient's family history includes a mother whodied at age 45 due to breast cancer. As soon as the DCE-MRI study isavailable in the hospital PACS system, the CAS algorithm is run on thecase in the background. A computer-aided detection algorithm identifiesa breast lesion at the left breast of the patient. Then, acomputer-aided diagnosis (CADx) algorithm is executed to derive alikelihood score (e.g., between 0 and 100) for malignancy, where thehigher likelihood scores correspond to a greater probability ofmalignancy. If the likelihood score is, e.g., between 0 and 20 orbetween 80 and 100, the stratification score is “easy”; if the scorebetween 20-30 or 70-80 the stratification score is “moderate”, and thescore is “difficult” if the CADx algorithm output is 30-70. Bothcomputer-aided detection and computer-aided diagnosis algorithms can beemployed (e.g., by performing segmentation of the lesion, featureextraction based the image and segmentation boundary, and a classifierto calculate a likelihood score).

To further this example, the CAS algorithm also uses demographics andother non-image-based information related to the patients. For example,in the above example, the woman has a family history of breast cancer,so even though the stratification score is “easy”, the score might beelevated to be “moderate” due to this extra clinical information, whichin turn may cause the case to be assigned to more experienced physiciansor double-read as a consequence.

The innovation has been described with reference to several embodiments.Modifications and alterations may occur to others upon reading andunderstanding the preceding detailed description. It is intended thatthe innovation be construed as including all such modifications andalterations insofar as they come within the scope of the appended claimsor the equivalents thereof.

1. A method of ranking patient cases according to diagnosis difficultylevel, comprising: for each of a plurality of patient cases: retrievingfrom a database an image study for a patient; identifying an abnormalityin a patient image included in the image study; analyzing patientdemographic and clinical information; calculating a computer-aidedstratification score for the patient case as a function of theidentified abnormality and the patient demographic and clinicalinformation; and outputting a ranked list of the patient cases accordingto the respective stratification score assigned to each patient casewherein the method further comprises: storing a plurality ofpreviously-diagnosed patient eases in the database; evaluatinghistorical diagnosis accuracy for the plurality of previously-diagnosedpatient cases; executing a classifier that generates an accuracy scoreindicative of the diagnosis accuracy for each of a plurality of types ofpatient cases; receiving information describing a current patient casetype; and generating a stratification score for the current patient casebased on the type of the current patient case and the accuracy score forthe type of the current patient case.
 2. (canceled)
 3. The methodaccording to claim 1, wherein each patient case is assigned to aclinician as a function of stratification score and clinicianexperience.
 4. (canceled)
 5. The method according to claim 1, furthercomprising, for each patient case: receiving lesion type and locationinformation for a lesion in the patient image; segmenting the image atthe lesion location to generate a first lesion outline; computing one ormore lesion parameters from the first lesion outline; computing one ormore alternative lesion outlines; computing one or more lesionparameters from the one or more alternative lesion outlines; andcomputing the stratification score as a function of lesion parametervariances between the lesion outline and the one or more alternativelesion outlines.
 6. The method according to claim 5, wherein computingone or more alternative lesion outlines comprises one of: randomlysampling image data from a neighborhood with a set distribution aboutthe lesion location; and employing a plurality of different segmentationalgorithms to segment the image at the lesion location.
 7. The methodaccording to claim 5, wherein the lesion parameters comprise one or moreof: lesion surface area; lesion volume; long axis of the lesion; andshort axis of the lesion.
 8. The method according to claim 1, furthercomprising: for each patient case: receiving lesion type and locationinformation for a lesion in the patient image; executing acomputer-aided diagnostic (CADx) technique on one or more of the lesionin the patient image, and patient demographic and clinical information,and determining a probability of malignancy of the lesion; deriving thestratification score for the patient case form the probability ofmalignancy; and adjusting a clinician workflow as a function ofstratification scores for of the plurality of patient cases.
 9. Themethod according to claim 1, further comprising flagging at least onepatient case, as a function of the stratification score for the at leastone patient case, as being candidate for review by at least tworeviewers.
 10. The method according to claim 1, further comprisingflagging at least one patient case, as a function of the stratificationscore for the at least one patient case, as being candidate for academicuse as a teaching case.
 11. (canceled)
 12. A system that facilitatesranking patient cases according to difficulty level, comprising: acomputer-aided stratification module comprising a processor adapted to,for each of a plurality of patient cases: retrieve from a database animage study for a patient; identify an abnormality in a patient imageincluded in the image study; analyze patient demographic and clinicalinformation; calculate a computer-aided stratification score for thepatient case as a function of the identified abnormality and the patientdemographic and clinical information; and output a ranked list of thepatient cases according to the respective stratification score assignedto each patient case wherein the system further comprises: a computerreadable medium that stores a plurality of previously-diagnosed patientcases; and wherein tin or is further configured to: evaluate historicaldiagnosis accuracy for the plurality of previously-diagnosed patientcases: execute a classifier that generates an accuracy score indicativeof the diagnosis accuracy for each of a plurality of types of patientcases; receive information describing a current patient case type; andgenerate the stratification score for the current patient case based onthe type of the current patient case and the accuracy score for the typeof current patient case.
 13. (canceled)
 14. The system according toclaim 12, wherein the abnormality is a lesion and the stratificationscore is calculated at least in part by segmenting the lesion image. 15.(canceled)
 16. The system according to claim 12, wherein the processoris further configured to, for each patient case: receive lesion type andlocation information for a lesion in the patient image; segment theimage at the lesion location to generate a first lesion outline; computeone or more lesion parameters from the first lesion outline; compute oneor more alternative lesion outlines; compute one or more lesionparameters from the one or more alternative lesion outlines; and computethe stratification score as a function of lesion parameter variancesbetween the lesion outline and the one or more alternative lesionoutlines.
 17. The system according to claim 16, wherein computing one ormore alternative lesion outlines comprises one of: randomly samplingimage data from a neighborhood with a set distribution about the lesionlocation; and employing a plurality of different segmentation algorithmsto segment the image at the lesion location.
 18. The system according toclaim 16, wherein the lesion parameters comprise one or more of: lesionsurface area; lesion volume; long axis of the lesion; and short axis ofthe lesion.
 19. The system according to claim 12, wherein the processoris further configured to: for each patient case: receive lesion type andlocation information for a lesion in the patient image; execute acomputer-aided diagnostic (CADx) technique on one or more of the lesionin the patient image, and patient demographic and clinical information,and determining a probability of malignancy of the lesion; derive thestratification score for the patient case form the probability ofmalignancy; and adjust a clinician workflow as a function ofstratification scores for of the plurality of patient cases. 20.(canceled)
 21. (canceled)
 22. A computer-readable medium having storedthereon computer-executable instructions for ranking patient casesaccording to diagnosis difficulty level, the instructions comprising:for each of a plurality of patient cases: retrieving from a database animage study for a patient; identifying an abnormality in a patient imageincluded in the image study; analyzing patient demographic and clinicalinformation; calculating and assigning a computer-aided stratificationscore for the patient case as a function of the identity of theabnormality and the patient demographic and clinical information; andoutputting a ranked list of the patient cases according to therespective stratification score assigned to each patient case whereinthe instructions further comprise: storing a plurality ofpreviously-diagnosed patient cases in the database; evaluatinghistorical diagnosis accuracy for the plurality of previously-diagnosedpatient cases; executing a classifier that generates an accuracy scoreindicative of the diagnosis accuracy for each of a plurality of types ofpatient cases; receiving information describing a current patient casetype; and generating a stratification score for the current patient casebased on the type of the current patient case and the accuracy score forthe type of the current patient case.